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1.  Introduction 

A  Javelin  trainer  called  the  EPBST  (Enhanced  Producibility  Basic  Skills  Trainer)  displays  a  terrain  scene  to 
the  gunner  in  either  visible  or  infrared  spectrum,  to  simulate  the  operation  of  the  Javelin  weapon  system.  One  task 
in  the  construction  of  these  terrains  is  the  stitching  together  of  a  series  of  infrared  images,  so  that  the  resulting  larger 
infrared  image  covers  and  lines  up  with  a  corresponding  visible  spectrum  photograph.  A  software  program  called 
Overlay  has  been  developed  to  help  perform  this  task,  and  is  the  focus  of  this  report. 

2.  Input  Data  and  Layer  Structure 

The  input  data  to  Overlay  is  a  single  visible  spectrum  image  together  with  a  series  of  up  to  255  infrared 
images.  The  visible  spectrum  image  is  stored  as  an  RGB,  24-bits-per-pixel  image  in  one  of  three  standard  image 
formats:  JPG,  PNG,  or  BMP.  The  infrared  images  are  gray-scale  images  which  can  be  stored  either  as  (1)  8-bits- 
per-pixel  in  one  of  three  standard  formats,  or  (2)  12-bits-per-pixel  in  the  original  FLR  format  that  is  produced  by 
sampling  data  from  the  ITAS  sensor.  Thus  when  using  IR  data  captured  from  the  ITAS  sensor,  the  images  may  first 
be  reduced  to  8-bits-per-pixel  and  stored  in  a  standard  image  format,  or  they  may  be  input  directly  into  Overlay,  in 
which  case  they  will  be  reduced  to  8-bits  within  Overlay  as  each  is  opened. 

The  input  images  are  called  layers,  and  are  numbered  in  the  order  in  which  they  were  opened.  The  visible 
spectrum  image  is  always  opened  first,  becoming  layer  0,  and  the  IR  layers  are  numbered  1,  2, ...,  N.  Typically  the 
IR  images  are  captured  in  an  up-and-down  or  back-and-forth  fashion,  and  follow  a  corresponding  filename 
convention.  Thus  if  they  are  opened  in  alphabetical  filename  order,  the  resulting  layers  should  progress  across  the 
visible  spectrum  image,  for  example  with  the  first  few  IR  layers  covering  the  left  edge  of  the  visible  spectrum 
image,  then  the  next  few  covering  a  column  of  imagery  just  to  the  right,  and  so  on.  It  is  not  necessary  to  open  and 
establish  the  complete  set  of  IR  layers  at  the  beginnning  of  the  workflow.  Instead,  the  first  few  IR  layers  may  be 


opened  and  manipulated  until  they  are  reasonably  well  lined  up  with  layer  0  and  each  other;  then  some  additional  IR 
layers  may  be  opened  and  manipulated,  and  so  on. 


Figure  1:  Overlay  Main  Window 


The  first  step  after  starting  up  the  Overlay  program  is  to  open  the  layer  0  image,  either  by  using  the  Open 
Visible  file  menu  command,  or  by  selecting  one  of  the  recently  opened  images  listed  at  the  bottom  of  the  File  menu. 
Assume  that  we  are  starting  a  new  project.  After  selecting  an  image  using  the  Open  Visible  command,  the  image 
appears  in  the  left  pane  of  the  Overlay  main  window,  and  the  right  pane  is  blank. 

There  are  two  File  menu  commands  for  opening  IR  layers.  Open  IR  and  Browse  IR,  each  of  which  brings 
up  a  standard  Windows  file  open  dialog  box: 

(1)  With  the  Open  IR  command,  only  one  file  may  be  selected,  which  is  then  opened  as  a  new  IR  layer,  using 
the  next  available  layer  number.  The  IR  image  is  displayed  in  the  right  pane  alongside  the  visible  spectrum 
image  in  the  left.  (See  Figure  1 .)  The  new  layer  may  be  “placed”  onto  the  stack  of  previously  displayed  images 
in  the  left  pane,  either  by  dragging  and  dropping  it  there,  or  by  point  measurement  and  fitting.  The  first  method 
is  activated  by  pressing  down  the  third  toolbar  button  from  the  left,  the  Move  Image  button.  With  that  toolbar 
button  activated,  look  for  a  feature  visible  on  both  the  new  layer  (in  the  right  pane)  and  the  visible  spectrum 
image  or  a  previously  placed  IR  layer  in  the  left  pane.  When  such  a  correspondence  is  spotted,  press  the  left 
mouse  button  with  the  cursor  over  the  feature  in  the  right  pane,  and  hold  it  down  while  dragging  the  cursor  to 
the  corresponding  place  in  the  left  pane.  The  second  method  (point  measurement  and  fitting)  is  discussed  in 
section  3. 

(2)  With  the  Browse  IR  command,  multiple  files  may  be  selected  at  one  time,  which  are  then  displayed  in  the 
right  hand  pane  of  the  main  window,  using  a  grid  pattern  of  thumbnail  (reduced  size)  images.  These  thumbnail 
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IR  images  are  not  yet  regarded  as  layers  by  the  software;  they  become  layers  only  when  dragged  and  dropped 
from  the  right  hand  pane  onto  the  left.  The  drag/drop  procedure  is  similar  to  that  explained  for  the  single  image 
case  above.  The  size  and  arrangement  of  the  IR  thumbnails  may  be  customized  on  the  Settings  dialog  box.  The 
set  of  available  IR  images  displayed  as  thumbnails  may  be  altered  by  reselecting  the  Browse  IR  command.  A 
Browse  Mode  menu  item  (under  the  View  menu)  allows  the  user  to  toggle  on  and  off  the  display  of  the 
thumbnail  IR  images. 

IR  layers  placed  using  the  drag/drop  procedure  described  above  are  scaled  using  the  current  value  of  an  IR-to-visible 
scale  parameter.  If  the  IR  layer  is  too  small  or  large  in  relation  to  the  visible  spectrum  image,  use  the  Adjust  Scale 
dialog  to  adjust  the  scale.  It  is  not  necessary  or  worthwhile  to  try  to  get  the  scale  perfect,  since  the  final  positioning 
of  the  layers  will  be  accomplished  later  using  block  adjustment. 
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Figure  2:  Layer  List  Dialog 


The  Layer  List  dialog  box  displays  the  layers  in  order  by  layer  number.  It  displays  each  layer’s  filename 
(minus  the  extension),  its  on/off  status,  and  lock  status.  The  on/off  status  determines  whether  the  layer  is  shown  in 
the  stacked  display  in  the  left  pane.  It  may  be  toggled  by  double-clicking  on  a  row  in  the  dialog  box;  there  are  also 
menu  commands  for  All  IR  layers  ON  and  All  IR  layers  OFF.  The  lock  status  determines  whether  a  layer  is 
allowed  to  be  repositioned  during  block  adjustment;  layers  marked  with  an  'L'  are  locked  in  place  during  that 
command.  Clicking  the  Lock  button  toggles  the  lock  status  of  the  active  layer. 

The  active  layer  is  the  layer  currently  highlighted  in  the  layer  list.  It  is  also  the  layer  displayed  in  the  right 
pane  when  not  in  Browse  Mode.  In  Browse  Mode,  the  right  pane  indicates  the  active  layer  with  a  small  square 
blue  marker.  (Other  images  which  have  been  loaded  are  indicated  by  red  markers.)  When  the  Show  Layer  Outline 
menu  item  is  checked  (under  the  View  menu),  a  blue  outline  indicating  the  placement  of  the  active  layer  is  drawn  in 
the  left  pane.  A  layer  can  be  made  active  by  clicking  on  its  row  in  the  layer  list,  by  clicking  on  its  thumbnail  in  the 
Browse  display,  or  by  right-clicking  on  a  location  in  the  left  pane  where  it  is  displayed. 

The  active  layer  may  be  deleted  using  the  Delete  button  on  the  Layer  List  dialog  box.  This  command  does 
not  delete  any  files;  it  Just  closes  the  image  and  removes  it  from  the  list  of  layers. 

The  Layer  List  dialog  also  has  four  buttons  for  controlling  the  stack  ordering  of  the  layers.  As  new  layers 
are  opened  they  are  initially  placed  on  top  of  the  stack;  i.e.  when  displayed  in  the  left  pane  the  new  layer  initially 
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appears  on  top  of  all  other  images.  The  visible  spectrum  image  (layer  0)  always  remains  at  the  bottom  of  the  stack, 
but  the  IR  layers  may  be  shifted  up  and  down.  Two  of  the  buttons  shift  the  active  layer  up  or  down  one  level  in  the 
stack;  the  other  two  move  it  directly  to  the  top  or  bottom  (not  counting  layer  0). 

Another  control  affecting  display  of  the  layers  is  the  transparency  scrollbar  to  the  right  of  the  toolbar  in  the 
’  Overlay  main  window  (see  Figure  1).  When  the  scrollbar  control  is  in  its  default  position  to  the  far  right  of  the 

!  scrollbar  as  in  Figure  1 ,  the  IR  layers  are  displayed  fully  opaque  over  the  visible  spectrum  image  in  the  left  pane. 

Moving  the  control  to  the  left  makes  the  IR  layers  transparent,  the  visible  spectrum  showing  through  more  and  more 
the  further  the  control  is  moved  to  the  left. 

3.  Point  Measurement  and  the  Fit  Command 

IR  layers  can  be  roughly  placed  onto  the  visible  spectrum  image  using  the  drag/drop  procedure  discussed  in 
section  2,  but  precise  positioning  requires  the  measurement  of  points  and  use  of  the  commands  (Fit,  Adjust  Block) 
which  make  use  of  point  measurements.  Here  “measuring”  a  point  refers  to  clicking  the  mouse  with  the  cursor  over 
some  noticeable  feature  in  the  image.  The  idea  is  to  measure  the  same  feature  in  more  than  image,  providing  the 
software  with  information  about  how  the  images  line  up  with  one  another.  For  the  most  part,  the  user  needs  only 
measure  points  linking  layer  0  with  the  IR  layers;  measurements  linking  pairs  of  IR  layers  are  generated 
automatically  during  block  adjustment. 

To  measure  a  point,  make  sure  Browse  Mode  is  turned  off,  and  press  the  Measure  Point  button  (the 
second  button  from  the  left  on  the  main  window  toolbar).  The  Point  List  dialog  is  displayed  if  it  was  not  already;  it 
lists  the  points  which  have  been  measured  on  the  active  layer.  Two  sets  of  coordinates  for  the  point  may  be  present: 
(x,  y)  pixel  coordinates  on  layer  0  and  (x,  y)  pixel  coordinates  on  the  active  layer.  Prior  to  the  point  having  been 
measured  the  coordinates  are  blank.  Thus  in  the  example  shown  in  Figure  3,  one  point  (with  point  id  0)  has  been 
measured  on  both  layer  0  and  the  active  layer,  and  the  software  is  ready  for  the  user  to  measure  a  second  point  (with 
point  id  1).  If  the  user  now  clicks  the  left  mouse  button  in  either  of  the  image  display  panes,  that  click  will  be 
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Figure  3:  Point  List  Dialog 
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interpreted  as  a  measurement  of  point  1 ,  on  layer  0  if  clicked  in  the  left  pane,  or  the  active  layer  if  clicked  in  the 
right  pane.  Note  that  a  mouse  click  in  the  left  plane  is  treated  as  a  measurement  on  layer  0  even  if  other  layers  are 
displayed  on  top  of  layer  0  at  the  location  of  the  mouse  click. 

The  highlighted  point  in  the  Point  List  dialog  is  called  the  active  point.  It  is  displayed  over  the  images  as  a 
diagonal  cross  (x)  as  opposed  to  other  points  which  appear  as  a  plus  sign  (+).  It  is  the  active  point  that  is  measured 
when  a  mouse  click  occurs,  whether  or  not  it  has  been  measured  before.  If  previously  measured,  and  the  mouse 
click  is  far  from  the  previous  position,  a  warning  message  allows  the  user  to  cancel  prior  to  moving  the 
measurement  to  the  new  location. 

The  New  Point  button  activates  a  new  point;  i.e.  one  which  has  not  yet  been  measured  on  either  image. 

Measurements  may  be  deleted  using  the  Delete  Point(s)  command  (under  the  Commands  menu). 
Highlighting  multiple  points  in  the  Point  List  enables  this  command  to  delete  multiple  points  at  one  time. 

The  Grab  Point  command,  activated  using  the  leftmost  toolbar  button,  enables  the  user  to  activate  a  point 
by  a  mouse  click  in  the  image  display  panes;  the  closest  point  which  has  been  measured  on  the  active  layer  becomes 
the  active  point. 

The  Window  Center  command,  activated  using  the  fourth  toolbar  button  from  the  left,  allows  the  user  to 
center  the  left  and  right  image  display  panes  on  features  of  interest  using  mouse  clicks.  A  typical  sequence  in 
measuring  a  point  is  to  activate  the  Window  Center  command  and  click  on  in  the  left  and  right  panes  on  something 
that  appears  to  be  a  common  feature;  having  centered  the  feature,  hit  the  Zoom  In  button  one  or  more  times  to  make 
the  feature  more  visible;  then  hit  the  Measure  Point  button  and  measure  the  point  if  satisfied  that  a  common  feature 
is  identifiable. 

The  Fit  command  may  be  used  to  update  the  placement  of  the  active  layer,  based  on  that  layer's  point 
measurements.  An  earlier  version  of  the  Overlay  program  did  not  contain  an  Adjust  Block  command,  and  relied 
entirely  on  the  Fit  command.  Because  the  Fit  command  adjusts  only  one  layer,  and  adjusts  that  layer  only  with 
regard  to  layer  0  and  not  the  other  IR  layers,  it  was  difficult  to  get  the  various  IR  layers  to  line  up  with  one  another, 
and  eliminate  visible  seams  between  the  IR  layers.  The  Adjust  Block  command,  as  explained  in  the  next  section, 
adjusts  the  placement  of  multiple  IR  layers  at  the  same  time.  It  eliminates  the  need  to  measure  several  points  linking 
each  IR  layer  to  layer  0;  it  is  often  enough  to  measure  just  a  few  such  points,  possibly  even  skipping  some  of  the 
interior  IR  layers  entirely,  and  then  allow  the  software  to  generate  tie  points  between  the  IR  layers  to  tie  everything 
together. 

4.  Block  Adjustment 

The  Adjust  Block  command  involves  a  set  of  layers,  called  a  block.  The  following  rules  determine  which 
IR  layers  are  included  in  the  block;  to  be  included  a  layer  must 

(1)  have  been  “placed”  onto  the  visible  spectrum  image  (by  the  drag/drop  procedure  or  the  Fit  command), 

(2)  be  toggled  “on”  for  display  in  the  left  pane,  and 

(3)  overlap  at  least  one  other  IR  layer  satisfying  (1)  and  (2),  and  either  the  layer  itself  or  one  that  it  overlaps 

must  be  unlocked. 
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When  the  Adjust  Block  dialog  is  displayed  (by  selecting  this  item 
from  the  Dialogs  menu),  the  rules  above  are  applied  and  the  layers 
found  to  be  in  the  block  are  listed  on  the  dialog  box  (see  Figure  4). 

The  Update  button  beneath  the  layer  list  is  used  to  reapply  the  rules 
after  making  changes  which  would  affect  the  block.  For  example,  the 
user  may  decide  to  lock  or  unlock  some  layers  after  displaying  the 
dialog;  hitting  the  Update  button  will  reapply  the  rules  to  create  a 
new  list  of  block  layers. 

In  the  example  shown  in  Figure  4,  seven  IR  layers  satisfied 
the  rules  and  were  included  in  the  block,  the  first  four  locked  and  the 
last  three  unlocked.  This  situation  might  arise  after  an  earlier  block 
adjustment  had  produced  a  satisfactory  placement  of  layers  1  through 
4.  The  current  block  adjustment  will  leave  those  layers  in  their 
current  positions,  but  use  their  positioning  in  adjusting  the  placement 
of  the  other  three  layers  (7,  8,  and  9). 

The  adjustment  depends  on  tie  points,  which  are  point 
measurements  like  those  discussed  in  section  3.  Each  tie  point 
consists  of  the  measurement  of  a  point  on  two  layers  of  the  block; 
here  we  regard  layer  0  as  a  block  layer  and  treat  it  as  being  locked. 

The  Tie  Points  list  on  the  dialog  displays  pairs  of  layer  numbers  in  the 
first  two  columns;  the  third  number  in  a  given  row  is  the  number  of  tie 
points  measured  on  those  two  layers.  The  rules  for  which  pairs  of 

layers  appear  in  this  list  are  similar  to  those  for  listing  the  layers  of  the  .  .  ,  i-  i 

o  ^  Figure  4:  Adjust  Block  dialog 

block;  a  pair  of  layers  is  listed  only  if  they  overlap  based  on  their 
current  placement,  and  only  if  at  least  one  of  them  is  unlocked.  In  the 

example  shown,  the  first  two  rows  indicate  that  two  points  have  been  measured  on  layer  0  and  layer  7,  and  likewise 
two  points  on  layer  0  and  layer  9.  These  are  the  only  four  points  measured  on  the  visible  spectrum  image  (layer  0) 
involved  in  the  current  block  adjustment.  (There  may  have  been  other  measurements  linking  layers  1  through  4  with 
layer  0,  but  these  are  not  included  since  both  layers  are  “locked.”)  The  other  tie  points,  those  linking  two  of  the  IR 
layers,  are  generally  produced  automatically  using  the  Add  or  Add  All  buttons,  although  they  can  also  be  measured 
using  the  mouse  as  described  in  section  4.  To  view  or  measure  tie  points  on  two  of  the  IR  layers,  highlight  a  row  in 
the  tie  point  list;  the  left  and  right  panes  will  then  display  the  corresponding  IR  images.  To  return  to  the  standard 
display  (IR  images  stacked  on  layer  0  in  the  left  pane),  highlight  any  layer  in  the  layer  list  of  the  Adjust  Block 
dialog,  or  highlight  layer  0  in  the  Layer  List  dialog. 

The  Add  button  generates  new  tie  points  for  the  highlighted  and  displayed  pair  of  IR  layers,  if  the  software 
is  able  to  find  corresponding  features  in  the  two  images.  The  search  is  performed  based  on  the  current  relative 
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placement  of  the  two  IR  layers;  when  the  software  is  unable  to  find  matches  it  may  help  to  adjust  the  layer 
placement  using  the  Move  Image  or  the  Fit  commands.  A  maximum  of  four  tie  points  are  added. 

The  Add  All  button  is  equivalent  to  applying  the  Add  button  for  each  pair  of  IR  layers  in  the  tie  point  list. 
The  Delete  button  deletes  tie  points  from  the  highlighted  pair  of  IR  layers.  The  typical  procedure  when  working 
with  new  IR  layers  in  a  block  adjustment  is  to  start  by  hitting  the  Add  All  button;  then  proceed  through  the  list 
highlighting  each  row  to  visually  examine  the  tie  points  generated.  Bad  tie  points  may  be  deleted  using  the  Delete 
Point(s)  command.  New  points  may  be  measured  using  the  Measure  Point  command  as  in  section  3  where  needed 
to  supplement  the  automatically  generated  tie  points. 

When  ready  to  perform  the  adjustment  it  is  a  good  idea  to  return  to  the  normal  stacked  display  in  the  left 
pane,  so  that  the  results  of  the  adjustment  can  be  viewed.  Hit  the  Adjust  button;  if  the  computation  is  successful  the 
unlocked  images  in  the  block  will  shift  around  in  the  left  pane  to  their  new  positions. 

5.  Output  Data 

Two  commands  are  used  to  save  results  in  Overlay:  Save  Data  and  Save  IR  (both  under  the  File  menu). 
The  Save  Data  command  writes  a  data  file  with  the  same  name  as  the  visible  spectrum  input  image,  but  having  a 
“.ov”  extension.  For  example  the  data  file  corresponding  to  image  file  cp3-L04.jpg  is  cp3-L04.ov.  This  data  file  is 
a  plain  text  file  viewable  in  any  text  editor.  It  stores  information  about  the  layers:  their  file  names,  placement 
relative  to  layer  0,  stack  order,  point  measurements,  and  brightness  adjustments.  When  a  file  is  opened  with  the 
Open  Visible  command  or  selection  from  the  recent  files  list,  Overlay  reads  the  corresponding  data  file  if  it  is 
present,  and  reconstructs  the  layers  from  the  information  in  the  data  file. 

The  Save  IR  command  is  used  to  save  the  composite  IR  image  constructed  from  all  the  IR  layers  as  they 
are  currently  displayed  in  the  left  pane.  Typically  the  user  will  want  to  use  the  All  IR  Layers  ON  command  prior  to 
the  Save  IR  command,  since  only  what  is  displayed  is  saved  in  the  output  image. 
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